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ABSTRACT 

The  new  emphasis  on  Anti-Terrorism  and  Force  Protection  (AT/FP),  for  both  shore  and  sea  platform 
protection,  has  resulted  in  a  need  for  infrared  imager  design  and  evaluation  tools  which  demonstrate  field 
performance  against  U.S.  Navy  AT/FP  requirements.  In  the  design  of  infrared  imaging  systems  for  target 
acquisition,  a  discrimination  criterion  is  required  for  successful  sensor  realization.  It  characterizes  the 
difficulty  of  the  task  being  performed  by  the  observer  and  varies  for  different  target  sets.  This  criterion  is 
used  in  both  assessment  of  existing  infrared  sensor  and  in  the  design  of  new  conceptual  sensors. 

In  this  experiment,  we  collected  12  small  craft  signatures  (military  and  civilian)  in  the  visible  band  during  the 
day  and  the  LWIR  and  MWIR  spectra  in  both  the  day  and  the  night  environments.  These  signatures  were 
processed  to  determine  the  targets’  characteristic  dimension  and  contrast.  They  were  also  processed  to 
bandlimit  the  signature ’s  spatial  information  content  (simulating  longer  range)  and  a  perception  experiment 
was  performed  to  determine  the  task  difficulty  (N5o  and  V so).  The  results  are  presented  in  this  paper  and  can 
be  used  for  Navy  and  Coast  Guard  imaging  infrared  sensor  design  and  evaluation. 


INTRODUCTION 

The  development  of  a  credible  asymmetric  threat  to  maritime  and  Naval  forces  has  been  a  relatively  recent 
occurrence.  The  bombing  of  the  USS  Cole  and  the  attacks  on  the  World  Trade  Center  in  New  York 
significantly  altered  our  threat  paradigm.  As  the  Navy  and  the  Coast  Guard  adapt  to  this  change,  we  need  to 
develop  the  tools  to  enable  the  community  as  a  whole  to  make  informed  decisions  and  plans  with  regard  to 
sensor  performance  and  hence  the  development  of  requirements  and  then  investment  in  resources. 

We  met  with  the  acquisition  program  managers  for  Coast  Guard  and  to  discuss  modeling  needs  and  to  develop 
the  target  set  for  this  experiment.  Additionally,  we  planned  out  a  series  of  experiments  to  develop  the 
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capability  to  model  EO/IR  sensors  in  the  performance  of  typical  tasks  required  in  the  maritime  security 
environment.  This  experiment  is  the  first,  and  most  straightforward  in  that  series. 

As  an  initial  effort,  we  teamed  with  the  US  Army  Night  Vision  and  Electronic  Sensors  Directorate’s 
(NVESD)  Modeling  and  Simulation  Division  (MSD)  to  apply  the  existing  validated  models  (for  example, 
NVTherm  the  infrared  imager  performance  model)  to  new  target  sets  and  classes.  Initial  efforts  applied  the 
methodology  to  the  identification  of  small  arms,  side  arms  and  various  handheld  threat  objects.1"3  NVESD  has 
conducted  numerous  experiments  to  extend  the  models’  performance  to  the  urban  environment  also.  4,5  As 
this  effort  was  very  successful,  we  elected  to  continue  the  collaboration  and  develop  the  criteria  for  identifying 
maritime  small  craft. 

2.0  TARGET  ACQUISITION  MODELING  BACKGROUND 

The  field  performance  of  an  imager  is  currently  described  by  the  system  contrast  threshold  function  (CTF). 
This  is  the  human  eye  CTF  while  the  human  is  looking  through  the  imager.  The  CTF  is  the  visual  threshold  at 
which  a  sinusoidal  input  can  be  seen,  where  the  sinusoid  extends  across  the  field-of-view.  The  system  CTF  is 
still  too  difficult  to  measure  in  the  laboratory,  but  can  be  related  to  the  currently  measured  Minimum 
Resolvable  Temperature  Difference  (MRTD)  measurement.  The  sensor  CTF  has  been  shown  to  be  effective 
in  characterizing  the  target  acquisition  performance  of  infrared,  image  intensifier,  solid  state  television,  laser 
range-gated  imagers,  and  TeraHertz  imagers  [NVESD  models  are  available  from  SENSIAC  at 
https://www.sensiac.gatech.edu,  for  US  government  and  contractor  use  and  from  NVESD  for  Allied 
government  agencies].  In  this  paper,  we  focus  on  solid  state  television  daytime  performance  and  midwave 
infrared  night  performance. 

The  typical  procedure  for  calculating/predicting  field  range  performance  is  shown  in  figure  1.  A  military 
target  is  described  by  the  characteristic  dimension  (square  root  of  target  area)  in  meters,  source  contrast,  and 
task  discrimination  difficulty  (N50  or  V5o).  The  contrast  is  propagated  through  the  atmosphere  and  an  apparent 
contrast  is  determined  at  the  sensor.  The  intersection  of  sensor  system  CTF  and  the  apparent  target  contrast  is 
called  the  “limiting  frequency,”  or  the  highest  frequency  resolvable  by  the  observer  through  the  sensor  at  that 
particular  contrast.  In  the  process  known  as  ACQUIRE,  this  limiting  frequency  in  cycles  per  milliradian  is 
converted  to  “cycles  on  target”  through  multiplying  the  limiting  frequency  by  the  target  angular  subtense 
(characteristic  dimension  divided  by  range).  The  cycles  on  target  are  compared  to  the  N50,  or  discrimination 
criterion  sometimes  called  the  Johnson  criterion,  to  determine  the  probability.  The  N50  is  different  for 
detection,  recognition,  and  identification  and  really  depends  on  the  target  set.  It  is  a  characterization  of  the 
task  difficulty.  The  ratio  of  cycles  on  target  to  N50  is  input  to  the  target  transfer  probability  function  (TTPF)  to 
provide  a  probability.  This  process  is  performed  iteratively  for  various  ranges  and  the  probability  is  plotted  as 
a  function  of  range. 
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N/^o  V/V50 


Figure  1.  ACQUIRE  and  TTP  Process. 


The  Target  Task  Performance  (TTP)  approach6"9  replaces  the  ACQUIRE  approach  for  most  military  target 
applications.  The  TTP  approach  varies  from  the  ACQUIRE  approach  in  that  a  system’s  resolution  and 
sensitivity  is  accounted  for  by  integrating  the  system’s  CTF, 


[cycles] 


(1) 


where  the  ratio  of  the  target  contrast  to  the  system  CTF  is  considered  the  excess  contrast  that  the  human  can 
see  on  the  target.  The  limits  on  the  integral  start  and  end  where  the  target  contrast  intersects  the  system  CTF. 
This  integration  is  performed  twice,  once  on  the  horizontal  CTF  of  the  system  and  then  again  on  the  vertical 
CTF  of  the  system;  equation  1  is  assumed  to  be  separable  in  spatial  dimensions.  The  geometric  mean  of  the 
horizontal  and  vertical  results  is  then  calculated  to  determine  a  system  value.  The  resultant  is  compared  to  the 
discrimination  criterion,  V50,  to  determine  the  probability.  A  different  TTPF  is  used  for  the  ratio  of  V  to  V50 
than  is  used  in  the  ACQUIRE  process. 

The  TTPF  is  the  function  that  converts  the  ratio  of  N  to  N50  or  the  ratio  of  V  to  V50  into  a  probability.  The 
TTPF  takes  the  form  of 
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where  the  coefficient,  P,  for  the  TTP  is  1.51+0.24(V/V50).  The  coefficient  for  ACQUIRE  traditionally  is 
2.7+0.7(N/N50),  but  a  single  coefficient  value  of  3.8  gives  very  close  results.  The  traditional  ACQUIRE 
coefficient  has  been  used  in  many  sensor  specifications  and  wargames,  however,  a  great  deal  of  real  field  data 
has  been  shown  to  match  a  more  gradual  probability  curve  with  coefficient  equal  to  1.75+0.35(N/N50)  or  a 
single  coefficient  of  2.7. 

The  TTP  process  provides  a  much  more  accurate  prediction  of  field  performance  than  the  ACQUIRE  process 
as  has  been  demonstrated  in  numerous  recognition  and  identification  experiments.  Search  and  detection  is 
more  complicated  (under  clutter-limited  conditions)  and  uses  a  modified  metric  to  account  target  acquisition 
in  a  clutter-limited  realm. 


3.0  DESCRIPTION  OF  THE  EXPERIMENT 

The  experimental  approach  was  to  develop  a  representative  target  set  for  small  maritime  surface  craft  and 
collect  signatures  for  creating  a  target  identification  perception  experiment.  The  resolution  of  the  imagery 
would  vary  in  order  to  determine  the  resolution  necessary  to  accomplish  the  task  (identification  or  ID)  with 
particular  levels  of  certainty.  An  ensemble  of  trained  observers  would  then  participate  in  the  ID  experiment 
and  statistical  results  would  yield  the  data  necessary  to  derive  the  discrimination  criteria  (N50  or  V50)  for  this 
task. 


4.0  SIGNATURES/IMAGES  COLLECTION 

After  discussions  with  the  Naval  Expeditionary  Combat  Command  (NECC)  personnel,  a  set  of  targets  was 
selected  which  included  military  patrol  craft,  military  working  craft,  commercial  working  craft  and  pleasure 
craft.  These  craft  were  assembled  at  Naval  Surface  Warfare  Center  Dahlgren  Division  (NSWC-DD)  for  the 
signature  collection. 

The  collection  site  was  the  waterfront  area  looking  out  to  the  south  towards  the  Potomac  River.  Data 
collection  occurred  throughout  the  diurnal  cycle  for  the  mid- wave  infrared  (MWIR),  3-5  micrometer  band, 
and  the  long-wave  infrared  (LWIR),  8-12  micrometer  band.  Visible  band  imagery  was  collected  during 
daylight  hours  only.  The  small  craft  were  presented  to  the  sensors  with  12  different  aspect  angles  varying 
from  bow-on  every  30°  around  the  compass  rose. 

The  sensors  used  included  a  LWIR  radiometer,  a  MWIR  radiometer  and  a  color  solid  state  digital  camera. 
Specifically,  the  LWIR  sensor  was  an  Indigo  TVS  700.  The  MWIR  sensor  was  an  AVIO  TVS  8500.  And  the 
visible  sensor  was  a  Nikon  D100. 


5.0  IMAGE/SIGNATURE  PROCESSING  &  PREPARATION 

Due  to  resource  limitations  and  priorities,  the  MWIR  night  imagery  and  monochrome  visible  day  imagery 
were  processed  and  perception  experiments  in  DEC05.  The  results  from  this  initial  set  of  experiments  were 
reported  and  presented  at  the  Military  Sensing  Symposia  Passive  Sensors  Conference  in  FEB06.  In  June  of 
2006,  the  MWIR  day  imagery  and  color  imagery  were  processed  and  a  further  set  of  perception  experiments 
conducted  using  all  four  bands/environments.  The  LWIR  day  and  night  imagery  remain  to  be  processed  for 
further  experimentation  in  the  future. 


9-4 


RTO-MP-SCI-180 


UNCLASSIFIED/UNLIMITED 


UNCLASSIFIED/UNLIMITED 

Small  Craft  Identification  Discrimination  Criteria 
for  Maritime  Anti-Terrorism  and  Force  Protection 


A  total  of  576  signatures  were  processed  for  use  in  this  research,  consisting  of  12  watercraft  viewed  at  12 
aspect  angles  in  each  of  two  wavebands.  The  visible  and  MWIR  sensors  differed  in  field  of  view,  focal  plane 
array  resolution,  and  recorded  image  format.  The  processing  performed  to  prepare  imagery  for  the  perception 
experiment  is  described  separately  for  each  sensor.  This  section  will  also  describe  the  target-to-background 
contrast  and  characteristic  dimension  calculations  for  each  spectrum. 

The  MWIR  imagery,  captured  during  the  day  and  at  night  with  a  radiometrically  calibrated  Avio  TVS-8500 
sensor,  was  recorded  as  14-bit  greyscale  images  in  IRI  format  with  a  resolution  of  256x236  pixels.  Perception 
experiments  were  to  be  run  on  high  resolution  monitors  in  a  larger  format.  This  prevents  the  monitor  pixel 
MTF  from  limiting  performance.  Therefore,  a  bilinear  interpolation  routine  was  used  to  increase  the  pixel 
count  by  a  factor  a  two  in  each  dimension,  resulting  in  a  512x472  pixel  image.  Subsequently,  the  14-bit 
greyscale  values  were  contrast  stretched  and  converted  to  12-bits.  The  contrast  stretching  procedure  allowed 
5%  saturation  at  the  highest  and  lowest  levels.  The  resulting  512x472  12-bit  images  were  the  baseline  set  of 
pristine  MWIR  signature  imagery  used  for  the  perception  experiment.  These  processing  steps,  performed 
using  MATLAB  software,  are  illustrated  in  a  block  diagram  in  Figure  2.  A  single  aspect  angle  example  of  the 
baseline  12-boat  image  set  in  the  MWIR  spectrum  is  shown  in  Figures  3 A  and  3B,  for  the  night  and  day 
respectively. 


Figure  2  Image  processing  steps  performed  on  MWIR  imagery. 


Figure  3A  MWIR  Night  /  3B  MWIR  Day  Baseline  12  Boat  Image  Set  (Port  Aspect  Shown) 


Visible  Imagery 

The  visible  imagery,  captured  during  the  daytime  with  a  Nikon  D100  digital  camera,  were  recorded  as  a  24-bit 
color  images  (8  bits  per  color  channel)  in  JPEG  format  with  a  resolution  of  3008x2000  pixels.  Images  were 
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cropped  in  the  horizontal  dimension  to  2000x2000  pixels,  then  a  bilinear  down-sampling  process  resulted  in  a 
1000x1000  pixel  image.  Image  size  was  reduced  in  this  manner  in  order  to  be  closer  to  the  size  and  format  of 
the  MW  imagery  presented  in  the  experiment.  The  day  visible  imagery  was  presented  in  two  perception 
experiments.  The  first  series  were  monochrome  and  the  second  color.  There  is  not  currently  a  color  target 
acquisition  model  (in  NVESD’s  family  of  models),  so  only  the  monochrome  results  are  analyzed 
quantitatively  and  N5o/V5o’s  developed.  Subsequent  to  down-sampling,  the  color  image  was  converted  to  256- 
level  greyscale,  then  contrast  stretched  resulting  in  4096  grey  levels.  The  contrast  stretching  procedure 
allowed  5%  saturation  at  the  highest  and  lowest  levels.  These  1000x1000  pixel  12-bit  images  were  the 
baseline  set  of  pristine  imagery  for  the  visible  spectrum.  These  processing  steps,  performed  using  MATLAB 
software,  are  illustrated  in  a  block  diagram  in  Figure  4.  A  single  aspect  angle  example  of  the  baseline  12-boat 
image  set  in  the  visible  spectrum  is  shown  in  Figures  5 A  and  5B  for  monochrome  and  color  respectively. 


Figure  4  Image  processing  steps  performed  on  monochrome  visible  imagery. 
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Figure  5A  Visible  Monochrome,  5B  Visible  Color  Baseline  12  Boat  Image  Set  (Port  Aspect  Shown) 


Forming  Images  for  Experimental  Cells 

For  the  day  and  night  in  both  the  MWIR  and  visible  spectra,  the  baseline  imagery  was  assigned  to  six 
experimental  cells.  The  cells  were  balanced  such  that  each  cell  contained  48  images,  consisting  of  12 
different  boats  at  4  different  aspects.  Cells  were  also  balanced  such  that  target  aspects  were  represented  in 
equal  numbers  in  each  cell,  in  order  to  avoid  any  bias  resulting  from  specific  target  aspects  that  were  most 
easily  identifiable  (such  as  port  or  starboard  views).  Each  target/aspect  combination  was  represented  twice 
across  all  six  cells,  resulting  in  an  experimental  database  of  288  MWIR  day  images,  288  MWIR  night,  288 
monochrome  visible  images  and  288  color  images. 
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After  assignment  to  an  experimental  cell,  each  baseline  image  was  convolved  with  a  normalized  Gaussian  blur 
filter.  The  Gaussian  blur  kernel  is  defined  by 


blur  =  exp 


■n 


J 


(3) 


The  blur  parameter,  b,  defines  the  width,  in  pixels,  of  the  Gaussian  filter.  Since  the  resolution  of  the  baseline 
imagery  differed  for  each  spectrum,  two  sets  of  blur  parameters  were  defined  in  order  to  result  in 
approximately  the  same  level  of  perceived  difficulty.  The  selected  values  of  b  for  each  experimental  cell  are 
listed  in  Table  1.  Examples  of  the  resultant  MWIR  imagery  presented  during  the  perception  trial  are  shown  in 
Figure  6.  The  top  row  shows  the  baseline  imagery,  while  the  middle  and  bottom  rows  show  examples  of 
blurred  imagery  for  each  experimental  cell.  The  resultant  imagery  for  the  visible  spectrum  shows  a 
comparable  amount  of  blur  across  the  experimental  cells. 


Table  1.  Width  of  Gaussian  Blur  Kernel  for  Each  Experimental  Cell 


Experimental  Cell 

Visible  Blur  (pixels 

MWIR  Blur  (pixels) 

A 

15 

5 

B 

27 

11 

C 

39 

17 

D 

51 

23 

E 

63 

29 

F 

75 

35 
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Figure  6.  MWIR  Experimental  Imagery  After  Gaussian  Blur  Processing. 


This  blurring  operation  is  a  quantifiable  method  of  systematically  reducing  the  number  of  resolvable  cycles  on 
the  targets  in  each  experimental  cell.  Analyzed  in  the  spatial  frequency  domain,  the  Modulation  Transfer 
Function  (MTF)  of  the  Gaussian  blur  kernel  is  multiplied  by  the  sensor  MTF  and  display  MTF,  resulting  in  a 
modeled  system  MTF.  As  the  blur  parameter,  b,  is  increased,  the  system  MTF  is  contracted,  limiting  the 
spatial  frequency  information  in  the  final  image  presented  to  the  observer  during  the  perception  trial.  The 
system  MTF  is  then  used  in  the  analysis  to  calculate  the  number  of  resolvable  cycles  required  to  achieve  a 
50%  probability  of  identification. 


Target  Size,  Target  Contrast  and  Delta  T  Calculations 

For  each  baseline  (pre-blur)  image,  the  target  was  segmented  from  the  background,  creating  a  target  mask 
with  pixel  values  of  1  for  all  target  areas  and  pixel  values  of  0  for  all  background  areas.  Target  area  was 
defined  as  any  pixels  containing  any  part  of  the  boat,  including  masts,  visible  ropes,  windows  which  contained 
glass,  and  visible  operators.  Disturbed  water  and  target-related  reflections  on  the  water  were  not  considered 
part  of  the  target  signature.  Characteristic  dimension  is  a  one-dimensional  quantity  calculated  as  the  square 
root  of  the  projected  target  area  in  display  pixels.  Where  target  contrast  is  defined  as: 


Contrast  = 


P bkgd 


)2 


2// 


scene 


(4) 


where  (itgt  and  pbkgd  are  the  average  target  pixel  value  and  the  average  background  pixel  value,  respectively. 
The  local  background  is  defined  as  the  area  adjacent  to,  but  not  including,  the  target  with  dimensions  the 
square  root  of  2  times  the  width  and  height  of  the  target.  The  standard  deviation  of  the  target  is  accounted  for 
by  otgt  and  the  result  is  normalized  by  twice  the  average  scene  pixel  value,  |iscene-  The  average  scene  value  is 
the  average  of  all  pixels  in  the  local  background  and  in  the  target  area.  As  shown  in  Table  2,  the  target 
characteristic  dimension  and  target  contrast  varied  by  only  a  few  percent  when  averaged  across  the  watercraft 
in  each  experimental  cell. 
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Table  2.  Target  Characteristic  Dimension  and  Contrast  for  Each  Cell 


MWIR  NIGHT  VISIBLE  DAY 


Blur  Cell 

C  o  n  tra  st 

L  scene 

T g  t  S  ize 

C  o  ntrast 

L  scene 

T g t  Size 

A 

0.179 

8.27  fL 

82  pix 

0.335 

4.7  5  fL 

176  p ix 

B 

0.181 

8.44  fL 

83  pix 

0.338 

4.7  8  fL 

174  p ix 

C 

0.177 

8.40  fL 

83  pix 

0.327 

4.64  fL 

178  pix 

D 

0.179 

8.27  fL 

82  pix 

0.335 

4.7  5  fL 

176  pix 

E 

0.181 

8.44  fL 

83  pix 

0.338 

4.78  fL 

174  p ix 

F 

0.177 

8.40  fL 

83  pix 

0.327 

4.64  fL 

178  p ix 

The  target  thermal  contrast  metric  (AT)  as  used  to  model  infrared  sensor  performance  in  NVThermlP  is  a 
root-sum-square  temperature  value,  calculated  by 

AT  =  +  KT  (5) 

where  all  parameters  are  analogous  to  the  contrast  equation,  but  in  units  of  degrees  Celsius,  rather  than 
greyscale  pixel  value.  The  AT  for  all  aspects  of  the  MWIR  night  baseline  watercraft  targets  ranged  from 
0.8°C  to  4.4°C,  with  an  overall  mean  AT  of  1.6°C.  The  AT  for  all  aspects  of  the  MWIR  day  baseline 
watercraft  targets  ranged  from  3.7°C  to  13.6°C,  with  an  overall  mean  AT  of  7.3°C. 


6.0  OBSERVER  TRAINING 

A  prototype  trainer  was  developed  for  use  in  preparing  the  subjects  to  perform  consistently  and  competently 
in  the  identification  of  the  boats  in  the  12-target  perception  experiments.  Signature  training  for  the  perception 
experiment  subjects  (Ss)  is  critical  to  achieving  a  low  noise  set  of  results.  The  training  was  designed  to  teach 
a  robust  set  of  cues  in  a  tutorial  format,  provide  interactive  practice  and  provide  exit  testing  to  assure  that, 
upon  completion,  Ss  could  achieve  a  basic  ID  score  of  at  least  95%  on  the  12-item  small  boat  target  set. 

The  signatures  of  the  boats  were  trained  in  3  distinct  training  groups  or  “Confusion  Sets,”  so  called  because 
the  members  of  each  group  are  similar  in  appearance  thus  easily  “confused.”  Teaching  the  boat  ID  using  this 
type  of  grouping  promotes  good  learning  of  the  critical  cues  of  the  signatures.  Three  confusion  sets  were 
established  based  upon  the  location  of  superstructure  found  on  the  craft.  Within  each  confusion  set,  the 
specific  distinguishing  identification  cues  for  each  boat  were  developed  based  on  general  characteristics  of 
boats  in  this  12-set  class.  The  boats,  their  respective  confusion  sets  are  shown  in  Table  3. 
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Table  3.  Small  Boats  Test  Trainer  Confusion  Sets 


Confusion  Set 

Boats  in  the  Set 

#1  Superstmcture  Forward 

Gatlin,  Monarch,  RCB3 

#2  Superstmcture  Amidships 

25’  Patrol,  34’  Patrol,  H920, 
Yellowtail 

#3  Superstmcture  Rear,  Open  or  No 
Superstmcture 

F470,  Fountain,  LCM8,  Manta, 
RHIB 

Cues  for  each  of  the  12  boats  were  developed  based  on  the  cue  1)  uniqueness,  2)  prominence,  and  3)  visibility. 
Cues  included  for  example  location,  size,  numbers  of  superstructure,  mast,  windshield,  out/inboard  engine(s), 
rails,  helm,  etc.  The  same  cues  were  used  for  thermal  ID  and  visible  ID.  Reference  images  used  in  the  trainer 
included  both  thermal  and  visible  images,  which  were  collected  simultaneously  in  the  field.  The  trainer  using 
MS  Office  Powerpoint  software.  The  curriculum  was  fashioned  after  the  ROCV  trainer  (Recognition  Of 
Combat  Vehicles),  an  NVESD-developed  trainer  in  use  by  the  joint  services  to  teach  combat  vehicle 
identification.  The  boat  trainer  was  self-paced  and  automated  to  assist  the  student  in  navigating  through  the 
curriculum.  This  consisted  of  the  student  first  studying  the  cues  presented  for  the  boats  in  one  of  the 
confusion  sets.  Second,  practicing  his  ID  of  the  boats  using  feedback  quizzes  provided  by  the  trainer.  The 
cues  were  studied  in  the  context  of  the  thermal  and  visible  images  then  that  study  was  applied  through  the 
feedback  quizzes.  The  process  involved  the  students  testing  themselves,  then  referring  back  to  the  cue 
presentations  when  ID  errors  were  made.  The  student  did  the  quiz  routine  until  by  his  own  judgment  he 
became  competent  and  confident  in  ID-ing  the  boats  in  that  confusion  set.  The  student  paced  through  all  three 
sets  in  this  manner  and  when  he  felt  ready  to  take  the  final  exit  test,  the  trainer  ported  him  out  of  the  trainer  to 
the  testing  coordinator.  The  TC  guided  him  thru  the  standard  perception  test  interface  to  take  the  final, 
qualifying,  exit  test. 

When  the  student  received  a  final  test  score  of  at  least  95%  on  the  12-boat  qualifying  exit  test,  he  was 
qualified  to  participate  in  the  experiment.  If  he  did  not  pass  the  test  then  he  simply  returned  to  the  trainer  for 
more  study  and  practice,  concentrating  on  those  boats  he  missed  in  the  test.  The  students  were  also 
encouraged  to  request  some  cue  type  help  from  the  perception  test  coordinator,  who  was  always  present  while 
the  students  trained. 

All  of  the  students  successfully  qualified  to  take  the  experiments.  The  training  time  required  by  the  group 
ranged  from  2  hours  to  10  hours,  with  an  average  of  ~ 6  hours.  Feedback  on  the  trainer  from  the  students 
indicated  they  felt  the  trainer  was  effective,  efficient  and  generally  enjoyable.  Samples  of  the  trainer  interface 
showing  screens  from  the  cues  section  and  the  practice  quiz  section  are  shown  below  in  Figures  7  and  8 
respectively. 
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Outboard  engines 


Monarch  Thermal  Cues 


Starboard; 

Mast  an  forward 
superstructure 

All  d*un 

Outboard  engines 


CS-1  Practice 


Port; 

Mast  on  forward 


superstructure 
*  Aft  d  o'.'h 


Answer 


Monarch  ! 


T?* 


Kit:  I’ 


Aft: 

Small  cenrter  masl 
Superstructure  rear 
enclosed 

Twin  outboard  w&tm 


Fore: 

Small  cenrter  masl 
slop  superslruclura  ■: 


Gatlin  .  Monarch  Ri'.’B'i 


Figure  7.  Small  Boat  Trainer 
Sample  Screen 


Figure  8  Small  Boat  Trainer  Sample 
Screen  for  Quizzes 


7.0  PERCEPTION  EXPERIMENT 

The  form  of  the  perception  experiment  was  a  12  alternative  fixed  choice  format.  The  fixed  choice  format  is 
very  is  the  accepted  method  of  performing  this  type  of  perception  experiment.  It  has  been  used  by  NVESD 
extensively  since  the  mid-1990s.  The  imagery  described  in  section  4.0  was  presented  to  the  observer  in 
random  order.  An  example  of  the  experimental  interface  is  shown  in  Figure  9. 


Figure  9.  Observer  Interface  Screen  Shot 

Each  observer  participated  in  the  visible  band  experiment  and  then  took  a  break.  After  the  break,  the  observer 
participated  in  the  MWIR  experiment.  In  all,  10  observers  were  trained  to  95%  or  greater  performance  and 
then  participated  in  the  perception  experiments. 
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8.0  RESULTS 

The  12  alternative  forced  choice  experimental  results  were  reduced  in  the  following  way.  First,  the 
probability  of  identification  for  each  level  of  blur  was  determined  by  taking  the  total  correct  answers  divided 
by  the  total  number  of  targets  presented  to  all  observers.  That  is,  the  average  probabilities  of  identification  for 
all  observers  were  averaged  to  provide  an  overall  probability  of  identification  for  the  ensemble  of  observers. 
The  standard  error  was  taken  as  the  standard  deviation  of  the  probabilities  corrected  for  the  number  of 
observers.  The  number  of  cycles  on  target  for  the  acquire  methodology  were  determined  for  each  level  of  blur 
and  the  probabilities  were  plotted  as  a  function  of  number  of  resolvable  cycles  on  target.  The  visible  target 
results  are  shown  in  figure  10  and  the  midwave  infrared  results  are  shown  in  figure  11. 


VIS-Daytime  Pid  vs  N 

Corrected  for  Chance  and  10%  Error 


N  (resolvable  cycles) 


Figure  10.  Visible  Monochrome  Day  Results. 


■jo 

ST 


MW  Night  P(id)  vs  N 

Corrected  for  Chance  and  10%  Error 


N50=4.0 

r2=0.97 

♦  Data 

—  Fit 

H - 1 - 1 - 

0  5  10 

- 1 — 

15 

N  (cyc/mrad) 

Figure  11  A.  MWIR  Night  Results. 
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MW  Day  P(id)  vs  N 

Corrected  for  Chance  and  10%  Error 


► 

N50=2.8 

r2=0.90 
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♦  Data 

—  Fit 
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Figure  11B.  MWIR  Night  Results. 

Note  that  the  fifty-percent  probability  of  small  boat  identification  (N50)  is  4.0  cycles  on  target  in  the  visible 
monochrome  band,  2.8  cycles  on  target  in  the  MWIR  day  and  4.0  cycles  on  target  in  the  MWIR  night.  The 
discrimination  task  in  the  visible  band  is  about  as  difficult  in  MWIR  night  and  visible  monochrome,  but 
significantly  easier  I  the  MWIR  day  case.  This  result  of  more  difficulty  in  the  visible  band  is  different  than 
for  most  other  targets,  such  as  tanks.  A  possible  explanation  is  that  tank  ID  uses  hot-spot  location  analysis, 
where  with  boats,  the  locations  of  hot  spots  (engine  and  exhaust  location)  do  not  provide  significant  benefit  in 
target  identification.  The  overall  shape  and  features  of  the  boat  are  the  primary  means  for  discrimination. 

In  terms  of  the  TTP  metric  (the  primary  metric  used  currently  by  the  Army  and  Marine  Corps),  the  fifty 
percent  probabilities  of  boat  identification  (V50)  is  14.0  cycles  on  target  in  the  monochrome  visible,  13.6  in  the 
MWIR  night  and  10.6  cycles  on  target  in  the  MWIR  day  case.  These  results  are  shown  graphically  in  Figures 
12  and  13A  and  13B  respectively.  The  results  are  also  shown  in  tabular  summary  format  in  Table  4. 

Table  4.  Results  Summary 


Discrimination 

Metric 

MWIR  (night) 

MWIR  (day) 

Visible  (day) 
greyscale 

Observers 

22 

10 

13 

DeltaT 

1.6  C 

7.3  C 

N/A 

Char  Dim 

3.9  m 

3.9  m 

3.9  m 

N50 

4.0 

2.8 

4.0 

V50 

13.6 

10.6 

14.0 
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VIS-Daytime  Pid  vs  V 

Corrected  for  Chance  and  10%  Error 


Figure  12.  TTP  Metric  Visible  Results. 


MW  Night  P(id)  vs  V 

Corrected  for  Chance  and  10%  Error 


Figure  13A.  TTP  Metric  MWIR  Night  Results. 
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Figure  13B.  TTP  Metric  MWIR  Day  Results 


9.0  DISCUSSION 

The  results  presented  in  the  previous  section  can  be  used  for  sensor  design  by  using  the  N50s  or  V50s  provided 
in  the  process  described  in  the  target  acquisition  section.  Coupling  the  criteria  with  the  acquisition  process 
will  allow  the  sensor  designer/analyzer  a  means  for  determining  sensor  performance  against  an  average  array 
of  boats  in  the  identification  process.  That  is,  sensors  can  be  designed  or  analyzed  against  an  ensemble  array 
of  boats  that  include  easy  to  discriminate  boats  and  difficult  to  discriminate  boats.  Using  a  90  percent 
probability  of  identification  requirement  will  provide  sound  sensor  design  rules.  However,  if  it  is  required 
that  a  particular  boat  be  identified,  further  experimentation  is  required. 

One  significant  result  is  that  the  difficulty  in  boat  identification  was  strongly  dependent  on  boat  aspect.  For 
the  MWIR  night  case,  the  N50  for  the  bow  and  stern  aspects  was  4.7  cycles  on  target  and  the  port  and 
starboard  aspects  was  2.4  cycles  on  target.  Identifying  boats  from  bow  and  stern  was  twice  as  difficult  as 
identifying  boats  from  broadside.  Some  discussion  between  the  authors  brought  up  the  issue  of  whether  a 
direct  bow  or  stern  aspect  should  be  a  consideration  since  any  small  angle  deviation  from  bow  or  stern 
provided  a  significant  increase  in  target  area  and  views  of  significant  features.  However,  a  good  point  is  that  if 
the  boat  were  approaching  (worst  case  for  anchored  high  value  asset)  or  fleeing  would  result  in  a  direct  bow  or 
stem  view.  The  MWIR  night  V50  for  bow/stern  was  17.9  cycles  on  target  and  the  MWIR  night  V50  for 
broadside  was  8.5  cycles  on  target.  The  visible  N50  for  bow/stem  was  5.3  cycles  on  target  and  broadside  was 
2.5  cycles  on  target.  Finally,  the  visible  V50  for  bow/stem  was  21.3  cycles  on  target  and  broadside  was  8.5 
cycles  on  target. 

Another  significant  result  was  the  lower  N50,  2.8,  for  the  MWIR  during  the  day  than  during  the  night,  4.0. 
This  is  somewhat  counter  to  conventional  reasoning  that  the  solar  contribution  should  be  treated  as  clutter 
enhancing.  In  this  result,  the  authors  believe  the  task  is  actually  made  easier  since  in  the  MWIR  signatures  are 
actually  a  combination  of  reflective  and  emissive.  At  night,  without  the  solar  reflection,  MWIR  appears  purely 
emissive.  The  day  signatures  actually  include  reflective  and  emissive  components,  and  hence  have  more 
information.  This  results  in  a  lower  task  difficulty  to  ID  vessels  in  the  day  time  MWIR. 


MW  Day  P(id)vs  V 
Corrected  for  Chance  and  10%  Error 
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10.0  CONCLUSIONS 

We  have  determined  the  discrimination  criteria,  N50  or  V50,  for  identifying  maritime  small  craft  in  both  the 
midwave  infrared  and  visible  bands.  These  are  numerical  metrics  describing  the  difficulty  of  identifying 
boats.  These  metrics  may  be  used  by  sensor  designers,  evaluators  or  wargamers  to  predict  the  range 
performance  or  probability  of  identification  for  a  given  existing  or  future  sensor  by  using  the  NVTherm  or 
SSCAM  models  available  from  NVESD. 

While  we  made  significant  progress  in  this  experiment,  the  series  of  experiments  will  continue  to  further 
determine  performance  criteria  for  other  tasks  inherent  in  Maritime  AT/FP.  We  also  have  the  remainder  of 
the  signature  data  from  this  collection  (LWIR  day  and  night)  to  process  and  include  in  perception 
experiments.  This  will  refine  the  picture  of  the  applicability  of  these  types  of  sensors. 

Additionally,  there  are  currently  two  other  data  collections/perception  experiment  efforts  in  the  planning 
stage.  The  first  is  related  to  determining  what  a  subject  is  doing  from  motion  video,  both  infrared  and  visible. 
The  second  is  related  to  determining  the  difficulty  in  detecting  and  classifying  swimmers  in  the  water.  Both 
of  these  experiments  will  expand  our  ability  to  model  and  predict  performance  of  AT/FP  tasks  in  the  maritime 
environment. 
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small  craft  identification  (ID)  task 

•  Give  sensor  engineers  a  calibrated  model  for 
evaluating  sensor  performance  for  small  craft  ID 


roach: 
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•  N5a  is  a  Discrimination  Criterion  -  a  measure  of  task 
difficulty,  similar  to  Johnson’s  Criteria 

•  N50  in  our  case  is  the  number  of  resolvable  cycles  across  the 
characteristic  target  dimension  corresponding  to  a  50% 
probability  of  identification  ( P1D) 

•  Target  Acquisition  (TA)  Model  allows  field  performance  to 
be  predicted  from  sensor  performance,  climate  conditions, 
and  target  characteristics. 

•  Performance  is  an  estimate  of  an  ensemble  of  observers 
against  and  ensemble  of  targets  &  specific  to  the  target  set 
used  to  empirically  measure  observer  performance. 

•  Limiting  frequency  (N50)  model  has  been  improving  since 
1950’s  and  has  been  successful  in  many  TA  tasks. 


Approach: 


Source  Contrast:  Csrc 
Char.  Dim.:  S=  ■s[Area^t 
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Range 


•  Collect  Signatures  of  Small  Craft  Target  Set 

•  Littoral  Environment 

•  Day  and  Night,  Sep-Oct  2005 

•  12  Watercraft  / 12  Aspects  covering  360° 

•  Radiometrically  Calibrated  Sensors 

•  Mid  Wave  IR:  Avio  TVS  8500 

•  Visible  Band:  Nikon  D 1 00 

•  Long  Wave  IR:  Indigo  TVS  700 

•  Process  Imagery 

•  Train  Human  Observers  on  Boat  ID 

•  Human  Perception  Testing 
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Sianature  Collection 
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Experimental  Imagery 


•  Optimize  Contrast 

-  Linear  histogram  stretch 


•  Resolvable  Cycles 

-  Apply  Gaussian  blur 

-  Known  spatial  frequency 
cut-off 

-  Six  different  levels  of  blur 
applied  to  imagery 


Full 

Resolution 


Experimental  Imagery 


BLUR  A 


BLUR  B 


Train  Observers  on 
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•  Observers  trained  on  cues  in  visible  and  MW  infrared 

•  Trained  to  a  level  of  95%  correct  ID  or  better  on 
“pristine”  imagery 

•  Self  paced  -  2  to  6  hours  each 


Human  Perception 


1 2  Alternative 
Forced  Choice 

Self-paced, 
Unlimited  time 

Presented  in  6  test 
cells  of  48  images 
each 

Test  cells  had 
different  levels  of 
blur. 


Results: 

MWIR  Night  vs.  Day 

N50  Calculations 
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MWIR  Night 
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MWIR  Day 
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MW  Night  P(id)  vs  N 

Corrected  for  Chance  and  10%  Error 
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•  Observers  performed  better  with 
daytime  imagery  than  with 
nighttime  imagery. 


•  Conventional  thinking  was  that 
MWIR  would  be  more  difficult 
during  the  day. 

Char  Tgt  Dim  =  3.9  m 


MW  Day  P(id)  vs  N 

Corrected  for  Chance  and  10%  Error 
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Results: 

Visible  Waveband 
Greyscale  vs.  Color 

N50  Calculation  and  PID  Comparison 
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Visible  Daytime  (Grey) 


Visible  Daytime  (Color 
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VIS-Daytime  Pid  vs  N 

Corrected  for  Chance  and  10%  Error 


•Visible  Daytime  is  comparable  in  06 

task  difficulty  to  MWIR  Night  io.5 

•Only  computed  for  Greyscale  °3 

Imagery,  there  is  no  defined  02 

Contrast  Threshold  Function  for  0.1 

Color  0 


Char  Tgt  Dim  =  3.9  m 

Compare  Probability  of  ID  of  color 
imagery  to  black  &  white  imagery 

Range  performance  improvement  with 
color 

Blur  levels  are  scalable  to  range  for  a 
specific  camera  system 


Blur  Level 


How  long  does  it  take  to 
identify  boats  in  Color 
Imagery  vs  Greyscale 
Imagery? 

Each  data  point  is  the  time 
to  correctly  identify  each 
boat  image  averaged  over 
all  observers. 

Overall  Weighted  Average: 

-  Greyscale:  3.1  sec 

-  Color:  2.9  sec 

ID  is  slightly  quicker  with 
Color 


Summary  of  Model 


•  This  Calibration  Allows  for  Modeling  the  Target 

Acquisition  Performance  of  Sensors  in  the  task  of  Boat  ID. 


Discrimination 

Metric 

MWIR 

(night) 

MWIR 

(day) 

Visible  (day) 
greyscale 

Observers 

22 

10 

13 

DeltaT 

1.6  C 

7.3  C 

N/A 

Char  Dim 

3.9  m 

3.9  m 

3.9  m 

N50 

4.0 

2.8 

4.0 

V50 

13.6 

10.6 

14.0 

•  Discrimination  Criterion  Was  Derived  for  Boat 
ID  for  MidWave  IR  night  time,  MidWave  IR 
day  time,  and  Visible  day  time  scenarios. 

•  Color  content  of  Visible  imagery  contains 
information  useful  to  Boat  ID. 

-  For  a  given  level  of  blur  (scalable  to  target  range), 
color  increased  performance  over  greyscale. 

-  10%  decrease  in  time  to  identify  color  images. 


